Papers with automatic evaluation protocol
PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain (2024.findings-acl)
Copied to clipboard
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, Baobao Chang
| Challenge: | a new multimodal decision-making benchmark evaluates the integrated capabilities of multimodal large language models. |
| Approach: | They propose a multimodal decision-making benchmark for evaluating MLLMs . they propose an automatic evaluation protocol to assess 10 prevalent ML models . |
| Outcome: | The proposed benchmark improves performance of multimodal large language models in three scenarios . the model is required to integrate multiple capabilities to make accurate decisions . |
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing web agents only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. |
| Approach: | They propose a large multimodal model-powered web agent that can complete user instructions end-to-end by interacting with real-world websites. |
| Outcome: | The proposed agent achieves 59.1% task success rate, surpassing both GPT-4 and WebVoyager setups. |
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automatic evaluation metrics for summarization are insensitive to factual inconsistencies. |
| Approach: | They propose an automatic evaluation protocol that detects factual inconsistencies in a model-generated summary. |
| Outcome: | QAGS has higher correlations with human judgments of factual consistency than other evaluation metrics. |
Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLMs as Your Coding Tutors (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on coding tutoring, but their capabilities in guiding users to solve complex tasks remain underexplored. |
| Approach: | They propose a novel agent workflow, Trace-and-Verify, which combines knowledge tracing to estimate a student’s knowledge state and turn-by-turn verification to ensure effective guidance toward task completion. |
| Outcome: | The proposed agent workflow achieves significantly higher success rates than existing tutoring agents. |
Improving Long-Context Translation via Self-Supervised Dual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models with long context windows suffer from catastrophic information distortion, undermining the strict faithfulness required for translation. |
| Approach: | They propose a self-supervised post-training framework that improves long-document translation reliability via round-trip consistency. |
| Outcome: | The proposed framework improves long-document translation reliability via round-trip consistency. |